Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add deferrable mode for Dataflow sensors #14

Conversation

e-galan
Copy link

@e-galan e-galan commented Feb 12, 2024

Introduces the deferrable mode for DataflowJobMessagesSensor and DataflowJobAutoScalingEventsSensor.

Changes:

  • New methods for DataflowJobMessagesSensor and DataflowJobAutoScalingEventsSensor;
  • New trigger classes: DataflowJobAutoScalingEventTrigger and DataflowJobMessagesTrigger;
  • New method for AsyncDataflowHook.;
  • Fixed some of the existing unit tests;
  • Added unit tests to cover the new code;
  • Additional documentation.

Questions:

  • It's not clear what to do with the example dags (the ones that go to the system tests) while 2 remaining Dataflow sensors are still waiting for the deferrable mode implementation.
  • By extension, it's not clear what to do with the .rst files that reference the example dags.
  • We have example Dataflow dags for several languages. Some of them are referenced in the .rst files, some of them are not. It's not clear if I should try and correct that, or I should focus only on the Python files.

@e-galan e-galan self-assigned this Feb 12, 2024
@e-galan e-galan marked this pull request as ready for review February 12, 2024 16:55
@e-galan e-galan changed the title Add deferrable mode to dataflow job auto scaling events sensor Add deferrable mode to for Dataflow sensors Feb 12, 2024
@e-galan e-galan changed the title Add deferrable mode to for Dataflow sensors Add deferrable mode for Dataflow sensors Feb 12, 2024
@e-galan e-galan closed this Feb 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants